United States Patent and Trademark Office 



UNITED STATES DEPARTMENT OF COMMERCE 
United States Patent and Trademark Office 
Address: COMMISSIONER FOR PATENTS 
P.O. Box 1450 

Alexandria, Virginia 22313-1450 
www.uspto.gov 



APPLICATION NO. 


FILING DATE 


FIRST NAMED INVENTOR 


ATTORNEY DOCKET NO. 


CONFIRMATION NO. 


09/702,288 


10/30/2000 


Wen-Yin Liu 


MS1-605US 


1430 



22801 ' 7590 12/31/2003 

LEE & HAYES PLLC 

421 W RIVERSIDE AVENUE SUITE 500 

SPOKANE, WA 99201 



EXAMINER 



TO, BAOQUOC N 



ART UNIT 



PAPER NUMBER- 



2172 

DATE MAILED: 12/31/2003 



Please find below and/or attached an Office communication concerning this application or proceeding. 



PTO-90C (Rev. 10/03) 



Office Action Summary 



Application No. 

09/702,288 



Examiner 

Baoquoc N To 



Applicant(s) 



LIU ET AL.8 




Art Unit 

2172 



- The MAILING DATE of this communication appears on the cover sheet with the correspondence address - 
Period for Reply 

A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) FROM 
THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1 .136(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If the period for reply specified above is less than thirty (30) days, a reply within the statutory minimum of thirty (30) days will be considered timely. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 133). 

- Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1 .704(b). 

Status 

1 )□ Responsive to communication(s) filed on . 

2a)D This action is FINAL. 2b)^ This action is non-final. 

3) D Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quay/e, 1935 CD. 1 1 , 453 O.G. 213. 

Disposition of Claims 

4) ^ Claim(s) 1-42 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

5) Q Claim(s) is/are allowed. 

6) ^ Claim(s) 1-42 is/are rejected. 

7) Q Claim(s) is/are objected to. 

8) D Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) D The specification is objected to by the Examiner. 

10) D The drawing(s) filed on is/are: a)D accepted or b)D objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1 .85(a). 
Replacement drawing sheet(s) including the correction is required if the drawing(s) is objected to. See 37 CFR 1.121(d). 

1 1) D The oath or declaration is objected to by the Examiner. Note the attached Office Action or form PTO-152. 
Priority under 35 U.S.C. §§ 119 and 120 

12) D Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 1 19(a)-(d) or (f). 

a)D All b)Q Some* c)D None of: 

1 .□ Certified copies of the priority documents have been received. 

2. D Certified copies of the priority documents have been received in Application No. . 

3. Q Copies of the certified copies of the priority documents have been received in this National Stage 

application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 

13) Q Acknowledgment is made of a claim for domestic priority under 35 U.S.C. § 119(e) (to a provisional application) 

since a specific reference was included in the first sentence of the specification or in an Application Data Sheet. 
37 CFR 1.78. 

a) □ The translation of the foreign language provisional application has been received. 

14) D Acknowledgment is made of a claim for domestic priority under 35 U.S.C. §§ 120 and/or 121 since a specific 

reference was included in the first sentence of the specification or in an Application Data Sheet. 37 CFR 1 .78. 



Attachment(s) 

1) El Notice of References Cited (PTO-892) 

2) O Notice of Draftsperson's Patent Drawing Review (PTO-948) 

3) O Information Disclosure Statement(s) (PTO-1449) Paper No(s) . 



4) O Interview Summary (PTO-413) Paper No(s). 

5) O Notice of Informal Patent Application (PTO-152) 

6) □ Other 



U.S. Patent and Trademark Office 
PTOL-326 (Rev. 11-03) 



Office Action Summary 



Part of Paper No. 11 




Application/Control Number: 09/702,288 Page 2 

Art Unit: 2172 

DETAILED ACTION 

1 . Claims 1-42 are pending in this application. 

Response to Arguments 

2. Applicant's arguments with respect to claims 10, 18, 25 and 32 have been 
considered but are moot in view of the new ground(s) of rejection. 

Claim Rejections - 35 USC § 103 
The following is a quotation of 35 U.S.C. 103(a) which forms the basis for 
all obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described 
as set forth in section 102 of this title, if the differences between the subject matter sought to 
be patented and the prior art are such that the subject matter as a whole would have been 
obvious at the time the invention was made, to a person having ordinary skill in the art to which 
said subject matter pertains. Patentability shall not be negatived by the manner in which the 
invention was made. 

3. Claims 1-9 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Atsushi et al. (A flexible content-based image retrieval system with 
combined scene description word, page 201-208, June 17-23, 1996, Multimedia 
Computing and System) in view of He etal. (US Patent No. 6,557,042 B1). 

Regarding on claims 1 and 9, Atsushi teaches a method comprising: 
identifying, in response to a search query (keyword), first multimedia 
objects having an associated keyword that matches a keyword in the search 
query (meaning of image) and second multimedia objects (location, color, shape 
of the segment) that have content features similar to those of the first multimedia 
objects (page 202, left column, lines 1 5-22); 

presenting the first and second multimedia objects to a user (page 104, 
left column, fig. 5). 
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monitoring feedback from the user as to which of the first and second 
multimedia objects are relevant to the search query; and annotating one or more 
of the multimedia objects, which are deemed relevant by the user, with keyword. 
He teaches the annotating process using button 332 of Fig 5 excepting for 
annotating the multimedia objects based on the user's feedback. He teaches, 
"there are may be a relationship defined between a "marketing" group and an 
"engineering group, such that user feedback from a user in the marketing group 
also modifies the corresponding segment score in the engineering group. This 
modification may be the same (e.g., increase both scores by two for positive 
feedback), or different (e.g., for e positive feedback increase the segment score 
in the user's group by two and increase the segment score in the related group 
by one)" (col. 9, lines 56-64). This teaches the increasing the score by two when 
there are positive feedback from the user means annotating the segment by two 
of the marketing and engineering group. Therefore, it would have been obvious 
to one ordinary skill in the art at the time of the invention was made to include the 
user's feedback to modify the scores to annotate the marketing and engineering 
group to the multimedia object segment to allow the retrieval of the same 
segment that relevant to these marketing and engineering. 

Regarding on claim 2, He teaches maintaining associations between the 
keywords and the multimedia objects, the associations being weighted to indicate 
how relevant the keywords are to the multimedia objects except for adjusting the 
weights of the associations based on the user's feedback (col. 9, lines 51-64). 
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Regarding on claim 3, He teaches the adjusting comprises increasing a 
weight of an association between the keyword and a particular multimedia object 
that is deemed relevant by the user (col. 9, lines 51-64). 

Regarding on claim 4, He teaches the adjusting comprises decreasing a 
weight of an association between the keyword and a particular multimedia object 
that is deemed irrelevant by the user (col. 9, lines 40-48). 

Regarding on claim 5, He teaches removing the keyword from the 
particular multimedia object in an event that the weight is less than a threshold 
value (col. 7, lines 55-60). 

Regarding on claim 6, He teaches training how the first and second 
multimedia objects are identified based on the user's feedback (col. 9, lines 23- 
56). 

Regarding on claim 7, He teaches refining the search to identify additional 
multimedia objects that contain content features similar to those of the 
multimedia objects indicated by the user as being relevant (col. 8, lines 55-60). 

Regarding on claim 8, Atsushi teaches the multimedia objects comprise 
one of digital images, video objects, and audio objects (col. 5, line 50). 
4. Claims 9-11, 13-26, 28-34 and 37-42 are rejected under 35 U.S.C. 103(a) 
as being unpatentable over He et al. (US. Patent No. 6,557,042). 

Regarding on claims 10, 17 and 32, He teaches a method comprising: 

Iteratively retrieving multimedia object from a data (the client-based 
browser also submits requests to the media server 104 for streaming data) (col. 
4, lines 9-11); 
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monitoring feedback from a user to whether the multimedia objects are 
relevant to a keyword in a search query (col. 9, lines 23-38); and 

He teaches the annotating process using button 332 of Fig 5 excepting for 
annotating the multimedia objects based on the user's feedback. He teaches, 
"there are may be a relationship defined between a "marketing" group and an 
"engineering group, such'that user feedback from a user in the marketing group 
also modifies the corresponding segment score in the engineering group. This 
modification may be the same (e.g., increase both scores by two for positive 
feedback), or different (e.g., for e positive feedback increase the segment score 
in the user's group by two and increase the segment score in the related group 
by one)" (col. 9, lines 56-64). This teaches the increasing the score by two when 
there are positive feedbacks from the user means annotating the segment by two 
of the marketing and engineering group. Therefore, it would have been obvious 
to one ordinary skill in the art at the time of the invention was made to include the 
user's feedback to modify the scores to annotate the marketing and engineering 
group to the multimedia object segment to allow the retrieval of the same 
segment that relevant to these marketing and engineering. 

Regarding on claim 1 1, He teaches the retrieving comprises using 
content-based information retrieval to retrieve the multimedia objects (col. 4, lines 
9-11). 

Regarding on claim 13, He teaches the monitoring comprises monitoring 
both feature-based relevance feedback and semantic-based relevance feedback 
(col. 9, lines 23-27). 
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Regarding on claim 14, He teaches the annotating is hidden from the user 
(col. 9, lines 22-37). 

Regarding on claim 15, He teaches the annotating comprises: 

in an event that a particular multimedia object is deemed relevant by the 
user and is not yet annotated with the keyword, adding the keyword to the 
particular multimedia object (col. 9, lines 49-64); and 

in an event that the particular multimedia object is deemed relevant by the 
user and is already annotated with the keyword, weakening an association 
between the keyword and the particular multimedia object (col. 9, lines 49-64). 

Regarding on claim 16, He teaches the annotating comprises: 

In an event that a particular multimedia object is deemed irrelevant by the 
user and is already annotated with the keyword, weakening an association 
between the keyword and the particular multimedia object (col. 9, lines 45-64). 

Regarding on claim 34, He teaches the search query comprises a content- 
based search query having one or more content features (col. 4, lines 9-11). 

Regarding on claim 37, He teaches the relevance feedback unit employs 
both feature-based relevance feedback and semantic-based relevance feedback 
(col. 9, lines 35-40). 

Regarding on claim 38, He teaches the search query comprise a keyword- 
based search query having at least one keyword; and in an event that a 
particular multimedia object is deemed relevant by the user and is not yet 
annotated with keyword, the annotation units adds the keyword to the particular 
multimedia object (col. 9, lines 23-39). 
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Regarding on claim 39, He teaches the search query comprises a 
keyword-based search query having at least one keyword; and in an event that a 
particular multimedia object is deemed relevant by the user and is already 
annotated with the keyword, the annotation unit strengthens an association 
between the keyword and the particular multimedia object (col. 9, lines 23-37). 

Regarding on claim 40, He teaches the search query comprises a 
keyword-based search query having at least one keyword; and in an event that a 
particular multimedia object is deemed irrelevant by the user and is already 
annotated with the keyword, weakening an association between the keyword and 
the particular multimedia object (col. 9, lines 23-38). 

Regarding on claim 41 , He teaches the search query comprises a 
keyword-based search query having at least one keyword; and in an event that a 
particular multimedia object is deemed irrelevant by the user and is already 
annotated with the keyword, removing the keyword from the particular multimedia 
object (col. 9, lines 23-38). 

Regarding on claim 42, He teaches the relevance feedback unit comprises 
a feedback analyzer to train the system based on the user's feedback (col. 9, 
lines 23-38). 

Regarding on claims 18 and 24, He teaches method comprising: 
retrieving multimedia objects according to a content-based retrieval 
process (col. 3, lines 56-57); 

presenting the multimedia object to a user (col. 3, lines 56-57); 
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monitoring feedback from the user as to which of the multimedia objects 
are relevant (col. 9, lines 49-64); and 

He does not explicitly teach annotating one or more of the multimedia 
objects based on the user's feedback. He teaches, "there are may be a 
relationship defined between a "marketing" group and an "engineering group, 
such that user feedback from a user in the marketing group also modifies the 
corresponding segment score in the engineering group. This modification may 
be the same (e.g., increase both scores by two for positive feedback), or different 
(e.g., for e positive feedback increase the segment score in the user's group by 
two and increase the segment score in the related group by one)" (col. 9, lines 
56-64). This teaches the increasing the score by two when there are positive 
feedbacks from the user means annotating the segment by two of the marketing 
and engineering group. Therefore, it would have been obvious to one ordinary 
skill in the art at the time of the invention was made to include the user's 
feedback to modify the scores to annotate the marketing and engineering group 
to the multimedia object segment to allow the retrieval of the same segment that 
relevant to these marketing and engineering. 

Regarding on claim 19, He teaches the monitoring comprises monitoring 
both feature-based relevance feedback and semantic-based relevance feedback 
(col. 9, lines 23-56). 

Regarding on claim 20, He teaches the annotating is hidden from the user 
(col. 9, lines 23-27). 

Regarding on claim 21, He teaches the annotating comprise: 
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in an event that a particular multimedia object is deemed relevant by the 
user and not yet annotated with the keyword, adding the keyword to the 
particular multimedia object (col. 9, lines 45-47); and 

in an event that the particular multimedia object is deemed relevant by the 
user and is already annotated with the keyword, strengthening an association 
between the keyword and the particular object (col. 9, lines 45-47). 

Regarding on claim 22, He teaches the annotating comprises: 

In an event that a particular multimedia object is deemed irrelevant by the 
user and is already annotated with the keyword, weakening an association 
between the keyword and the particular multimedia object (col. 9, lines 45-47). 

Regarding on claim 23, He teaches the annotating comprises: 

In an event that a particular multimedia object is deemed irrelevant by the 
user and is already annotated with the keyword, removing the keyword from the 
particular multimedia object (col. 9, lines 35-40). 

Regarding on claims 25 and 31 , He teaches a method comprising: 

retrieving a set of one or more multimedia objects for presentation to a 
user (col. 4, lines 9-11). 

adjusting the weights of the associations based on the user's feedback 
(col. 9, lines 49-56). 

monitoring feedback from the user as to which of the multimedia objects 
are relevant (col. 9, lines 49-56) 

He does not explicitly teach maintaining association between keywords 
and multimedia objects, the associations being weight to indicate how relevant 
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the keywords are to the multimedia objects. However, he teaches "alternatively, 
there may be relationship between different groups that result in summary 
information for multiple different groups being modified. There are may be a 
relationship defined between a "marketing" group and an "engineering group, 
such that user feedback from a user in the marketing group also modifies the 
corresponding segment score in the engineering group. This modification may 
be the same (e.g., increase both scores by two for positive feedback), or different 
(e.g., for e positive feedback increase the segment score in the user's group by 
two and increase the segment score in the related group by one)" (col. 9, lines 
56-64). This teaches the association between the segment from the user groups, 
this corresponding to the objects and keywords. Therefore, it would have been 
obvious to one ordinary skill in the art at the time of the invention was made to 
include the association of the multimedia segments with the different groups as 
to the multimedia objects with the keywords in order to allow efficient retrieval 
process for relevant results. 

Regarding on claim 26, He teaches the retrieving comprises using 
content-based information retrieval to retrieve the multimedia objects (col. 4, lines 
9-11). 

Regarding on claim 28, Morag teaches the monitoring comprises 
capturing both feature-based relevance feedback and semantic-based relevance 
feedback (col. 6, lines 52-55). 
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Regarding on claim 29, He teaches the adjusting comprises increasing the 
weights of the association between the keywords and the multimedia objects that 
are deemed relevant by the user (col. 9, lines 56-64). 

Regarding on claim 30, He teaches adjusting comprises decreasing the 
weights of the association between the keywords and the multimedia objects that 
are deemed irrelevant by the user (col. 9, lines 45-48). 
5. Claims 12, 27 and 35-36 is rejected under 35 U.S.C. 103(a) as being 
unpatentable over in He et al. (US Patent No. 6,557,042 B1) in view of Atsushi 
et al. (A flexible content-based image retrieval system with combined scene 
description word, page 201-208, June 17-23, 1996, Multimedia Computing and 
System). 

Regarding on claims 12, 27 and 35-36, He does not explicitly teach the 
retrieving comprises using both content-based information retrieval and 
semantic-based information retrieval to retrieve the multimedia objects. 
However, Atsushi teaches, "let us suppose that a user wants an image with blue 
sea and a white house. The sea segment in at lower and the white house is on 
the sea. You can see the state water in State transition model as shown in fig. 1, 
so the retrieval query for blue sea at lower is made by Conceptual keyword" 
(page 204, left column, lines 24-32). This teaches the searching method utilizing 
both keywords and feature. Therefore, it would have been obvious to one 
ordinary skill in the art at the time of the invention was made include both 
keyword and feature to allow the user to conduct a search more specific and 
retrieve the results as requested. 
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If attempts to reach the examiner by telephone are unsuccessful, the 
examiner's supervisor, John Breene can be reached at (703) 305-9790. 

Any inquiry of a general nature or relating to the status of this application 
or proceeding should be directed to the receptionist whose telephone number is 
(703) 305-3900. 
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